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(57) Abstract: A real-world 

representation system comprises 
a set of devices, each device being 
arranged to provide one or more 
real -world parameters, for example 
audio and visual characteristics. At 
least one of the devices is arranged to 
receive a real-world description in the 
form of an instruction set of a markup 
language and the devices are operated 
according to the description. General 
terms expressed in the language are 
interpreted by either a local server 
or a distributed browser to operate 
the devices to render the real -world 
experience to the use 
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DESCRIPTION 

A REAL-WORLD REPRESENTATION SYSTEM AND LANGUAGE 

5 This invention relates to a method of controlling a set of devices, a real- 

world representation system, a markup language enabled device and a 
markup language. 

In order to make the Internet accessible to a larger number of people 
HTML (HyperText Markup Language) was developed along with browsers that 

10 could read HTML files. HTML is a language that principally describes 
documents and links between documents. The browser reads the HTML code 
and displays the document comprised of the text and links on a PC screen. In 
order to extend the rather limited text only nature of HTML, JAVA was 
developed. JAVA is a programming language and a JAVA program (or applet 

15 as it is commonly referred to) is compiled from source to object code and is 
then addressed by an HTML link to produce graphics etc. in the right place on 
the screen as desired, i.e. an HTML document can include within it a link to a 
compiled JAVA applet. In due course, to further extend the information that 
can be transferred by the Internet, VRML was developed. VRML originally 

20 stood for Virtual Reality Markup Language, but the M is now more often 
referred to as standing for Modelling. VRML is a way of describing a three- 
dimensional place or object in terms of its geometry that can then be 
represented on a screen that is user navigable. 

However, none of these languages can even begin to describe a real- 

25 world experience nor render it for the end user. 

According to a first aspect of the present invention, there is provided a 
method of operating a set of devices comprising receiving a real-world 
description in the form of an instruction set of a markup language and 
operating said devices according to said description. 

30 The method advantageously further comprises distributing the 

description amongst the devices or reading the description at a local server. 
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The description may form part of a broadcast signal and/or it may relate to an 

activity that a user is undertaking. 

According to a second aspect of the present invention, there is provided 

a real-world representation system comprising a set of devices, each device 
5 arranged to provide one or more real-world parameters, at least one of said 

devices arranged to receive a real-world description in the form of an 

instruction set of a markup language, and said devices being operated 

according to said description. 

Owing to these two aspects of the invention, it is possible to provide and 
10 render a real-world experience. 

Preferably the devices are interconnected by a wireless network or by a 

powerline carrier network. 

According to a third aspect of the present invention, there is provided a 

markup language enabled device comprising receiving means for receiving a 
15 real-world description in the form of an instruction set of a markup language 

and adjusting means for adjusting one or more characteristics of the device 

according to said description. 

Owing to this aspect of the invention, it is possible to provide a device 

that can be utilised in a system for providing a real-world experience. 
20 Advantageously the receiving means of the device includes part of a 

distributed browser stored on a record carrier, the part interpreting the 

instruction set and communicating with the adjusting means. The browser can 

be arranged to interpret descriptions of a general type to generate specific 

parameter instructions for communicating to the adjusting means. 
25 According to a fourth aspect of the present invention, there is provided a 

markup language for describing real-world experiences comprising an 

instruction set including instructions interpretable by a markup enabled device 

to operate said device. 

Owing to this aspect of the invention, it possible to create instruction 
30 sets that correspond to real-world experiences, within the confines of a markup 

language, that can be used by enabled devices to render those experiences. 
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Such real-world experiences include visual, audio, olfactory and tactile 
sensations. 

The markup language describes experiences in the physical world. It 
allows the authoring of spatial, temporal and structural elements but also more 
5 ambient qualities such as mood, sensory factors and the dynamics of the 
space. 

Embodiments of the invention will now be described, by way of example 
only, with reference to the accompanying drawings, in which: 

Figure 1 is a schematic perspective view of a real-world representation 
10 system, 

Figure 2 is a perspective view of a markup language enabled device of 
the system of Figure 1 , 

Figure 3 is a flowchart of a method of controlling a set of devices, and 
Figure 4 is a flowchart of an alternative method of controlling a set of 
15 devices. 

In Figure 1 the real-world representation system comprises a set of 
devices including a display device 10, audio speakers 12, a lighting device 14, 
a heating device 16, walls 18 and the floor 20. These devices together 
contribute to make up the ambient environment, each device being arranged 

20 to provide one or more real-world parameters. For example, the lighting device 
14 contributes colour tones as well as to the luminance level. The devices may 
be electronic or they may be purely mechanical. The devices are 
interconnected by either a wireless network or a wired network such as a 
powerline carrier network or a conventional cable system such as RS232 

25 leads. 

In addition to the display device 10, the walls 18 and floor 20 are 
provided with display functionality. This can be achieved either by the walls 18 
and floor 20 being composed of fabric display material, or a centred ceiling 
projector can illuminate the visible portions of the walls 18 and the floor 20. It is 
30 also feasible to back light the walls 1 8 if the environment allows it. 

The system of Figure 1 is shown as being a room with a couch 22 for a 
user, the room being part of a private dwelling. However the room could be, for 
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example, a hotel room or part of an entertainment complex or could form part 
of a public space. 

At least one of the devices making up the real-world representation 
system is arranged to receive a real-world description in the form of an 
5 instruction set of a markup language, the devices being operated according to 
said description. This description could form part of a broadcast signal or it 
could be recalled from a local or remote store. The description could relate to 
an activity that a user is undertaking or could be triggered from a user interface 
or could trigger automatically from some predefined event, for example, a user 
entering a room. 

In a first embodiment (outlined in the flowchart of Figure 3) this 
description is distributed amongst the devices. Each markup language enabled 
device operates in principally the same manner. For example, the lighting 
device 14 (shown in more detail in Figure 2) has receiving means 24 for 
receiving the real-world description in the form of an instruction set of a 
markup language, the receiving means 24 including part of a distributed 
browser that interprets the instructions of the instruction set. The portion of the 
browser in the receiving means 24 communicates with adjusting means 26 that 
is arranged to adjust one or more parameters of the lighting device 14. For 
example, if the real-world description reads <FOREST>, <SUMMER>, 
<EVENING> then the browser part in the receiving means 24 interprets this 
into specific instructions relating to the colour tones and luminance level for the 
adjusting means 26 to adjust their levels accordingly. In this example the likely 
colour tone would be a pleasant green and the light level would be low but 
warm. The browser part interprets instructions of a general type to generate 
specific parameter adjustments. 

The description is received by all of the other parts of the distributed 
browser in each of the other devices in the real-world representation system. 
Each device interprets the instructions and adjusts its parameter configuration 
accordingly. The more enabled devices that are present in a particular 
environment the more realistic the end result will be. In particular the 
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operability of the walls 18 and floor 20 as display devices will determine how 
immersed in the real-world representation the user feels. 

In a second embodiment (outlined in the flowchart of Figure 4) the 
description is read at a local server, which can be a dedicated device or could 
be contained within the capability of a device that nominally has a different 
purpose. In this embodiment a browser or operating system present on the 
local server interprets the instructions of the real-world description and 
generates specific parameter adjustments for communicating to the relevant 
device. In this way devices currently available can be used without the 
requirement that they be modified or purpose built for use in the real-world 
representation system. 

In addition to the devices shown in Figure 1 , a wide variety of additional 
devices are possible to augment the user's experience. These include such 
things as robotic devices such as robot lamps, interactive art such as pictures 
and sculpture, animatronic furniture and animatronic electronic equipment. All 
of these can add in to the experience according to their functionality. 

The inclusion of the description in a broadcast signal allows the user to 
be immersed in an environment that is linked to a broadcast that he is 
receiving, for example a broadcast television signal. I.e. if the user is watching 
a film, then as the film moves through various scenes in turn these can be 
rendered by the set of devices making up the real-world representation 
system. If an underwater scene is showing on the television then the 
broadcast could include a description <WATER>, <COLD> and this will be 
interpreted by the relevant parts of the system to render the real-world 
experience to the user, turning the lights blue etc. The description need not be 
itself included in the broadcast signal, for example, a URL may be embedded 
in the signal and the description is communicated via a web link using the 
URL. 

The description can be received at all locations where there is a real- 
world representation system operating and each individual system will render 
the experience to the best of the functionality of the devices in the system. For 
example, a cinema may be so designed that it can render experiences through 
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devices other than the conventional display screen and audio system. This will 
allow films that are augmented with descriptions to provide additional 
immersion into the film for viewers at the cinema. 

The description can also relate directly to an activity that a user is 
undertaking, for example, reading a book. The individual pages of the book 
can trigger different descriptions being passed to the real-world representation 
system. Each description is linked by content to the particular theme or scene 
of the book at the respective point in the book. For example, the children's 
story "Little Red Riding Hood" has a number of discrete scenes within it, 
including a forest and the interior of a cottage, each of which can be rendered 
by the devices of the system in turn depending upon the place in the story that 
the reader has reached. A movement sensor may be employed to detect the 
turning of the pages, or the book may have inbuilt functionality that 
automatically detects which page the reader is looking at and triggers the 
appropriate description that is then rendered by the system. The triggering is 
ideally carried out via a wireless connection. 

The description may also be delivered to a user in a manner other than 
via a broadcast signal. For example, the description may be triggered from a 
physical carrier such as a postcard which is augmented to include a 
description (or a URL pointing to a description) in the form of an instruction set 
of the markup language. This description could relate to the view depicted on 
the postcard. The description could be stored on the postcard in the form of a 
barcode, embedded RF tag or other similar technology. When the user 
receives the postcard their real-world representation system is transformed to 
reflect the scene or experience on the postcard. A tag reader coil can be 
placed around the letterbox, so that when the postcard arrives the system in 
the user's house reacts according to the description embedded in the 
postcard. The description on the postcard need not relate to the postcard, but 
could equally relate to the sender of the postcard. 

Other methods of delivering the description to a user are possible. 
Local IR and RF systems can be utilised as carriers for descriptions. In the 
case of IR, this would be a standard or high end remote control that an 
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individual uses to control their consumer electronic devices around the home. 
The remote control is provided with, or has the ability to upload, descriptions or 
fragments of the markup language. 

A user can therefore operate the system in a relatively simple manner 
from a user interface such as a remote control, although equally an enabled 
PDA or other mobile terminal could be employed. The user can select an 
environment that they wish to be immersed in from preselected total 
descriptions or create a new environment from pre- or userdefined variables. 
Such a total description may, for example, be <CHINESE RESTAURANT>, 
which real-world experience is then rendered by the devices in the system. 
The real-worlds generated can be fantasy environments or they can be 
realistic. 

RF systems, such as Bluetooth are also suitable for transmitting and 
receiving descriptions. In this way descriptions can migrate by being stored on 
devices with Bluetooth capability.) Such devices would be PDAs, mobile 
phones, lap top computers etc. As these devices are carried by users from 
environment to environment, there exists the ability to pass descriptions 
between the mobile device and a local storage medium. In this way new 
descriptions are received by environments. 

The internet is also a route by which descriptions may be delivered to a 
user or directly to an environment. As the description is provided in a markup 
language format, descriptions and fragments of descriptions can easily be 
stored on servers for recall via a PC or suitably enabled digital TV. The 
descriptions can be updated and amended by the authors, thereby allowing a 
large and varied library of descriptions to be created, all of which can be easily 
accessed by a user. 

A user can also generate instruction sets to augment experiences such 
as films. In effect the user authors descriptions that can be added to the 
original content. In this way their enjoyment of the film is increased. The user 
adds a data track to a recordable format disc, for example CD+RW or 
DVD+RW, via a PC to create a set of effects to go with the various scenes of 
the story. When the film is played back in an enabled environment, an 
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enhanced experience of the film is provided. In addition to authoring their own 
description, a user can recall descriptions from a library of instruction sets, 
which may be provided, for example, via the Internet. 

The user can also operate the user interface to set limits on the 
5 operation of the devices in the system. For example, if the user wishes to set 
the volume of those devices with an audio capability to a specific range or set 
an upper limit on the volume then they can do so via the user interface. This 
prevents experiences becoming unpleasant for the user. The level of the light 
and the rate of any change in light intensity are also things that the user can 

10 control. All of the parameters of the system can be user defined. 

In addition to the examples described above, which can be 
characterised as generally involving a "passive" user, interactive applications 
of the system are possible. An example of this is a game that a user may play, 
via a PC or games console for example. If the game is augmented with a 

15 description relating to the scenario and/or locations of the game world, then as 
the user navigates the game world the environment around them changes 
accordingly. 

The description of the real-world experiences is provided by a markup 
language that communicates a description of physical environments and the 

20 objects within them, their relationship to the user, each other and to the 
physical space. Within a location that is enabled to produce a real-world 
experience, the instruction set of the markup language is interpreted by a 
device or devices to render the experience. Each device that is enabled 
contains a component that interprets the instruction set to the best of its 

25 capability. 

The language contains a wide range of states that can be rendered by 
the devices in a real-world representation system. Such states relate to:- 

• Image display - specific images, streamed video 

• Audio - music, sound effects, voice 
30 • Mood - emotional, ambient, animated 

• Light - levels (relative/absolute), moods, colours, position, focus 

• User display and input - feedback, menu display 
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Time - time of day, season 

Location - absolute, fantasy, generic type. 

Smell - background, responsive 



Motion - robot devices 



5 



Animatronic - movable furniture 



The language has the ability to be extended to cover other states and 
forms of representation as desired. 

Any information that relates to a physical experience can be expressed 
in this markup language. To create instruction sets in the markup language an 
10 author can write directly into the language or a computer program for authoring 
an instruction set can be used. The authoring program can take data input in 
the form of, for example, text or video and can generate a set of instructions 
comprising a subset of the markup language that when passed to a real-world 
representation system will allow the devices in that system to render the 
15 experience that corresponds to that of the text or video. 

The language is XML compliant, XML being a metalanguage that 
describes the standard structure for new markup languges. 
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CLAIMS 

1 . A method of operating a set of devices comprising receiving a 
real-world description in the form of an instruction set of a markup language 

5 and operating said devices according to said description. 

2. A method according to claim 1, and further comprising 
distributing said description amongst said devices. 

10 3. A method according to claim 1, and further comprising reading 

said description at a local server. 

4. A method according to any preceding claim, wherein said 
description forms part of a broadcast signal. 

15 

5. A method according to any preceding claim, wherein said 
description relates to an activity that a user is undertaking. 

6. A method according to any one of claims 1 to 4, wherein said 
20 description is triggered from a user interface. 

7. A method according to claim 6, wherein said interface is a book. 

8. A method according to claim 6 or 7, wherein said triggering is 
25 carried out via a wireless connection. 

9. A method according to any one of claims 1 to 4, wherein said 
description is triggered from a physical carrier. 

30 10. A real-world representation system comprising a set of devices, 

each device arranged to provide one or more real-world parameters, at least 
one of said devices arranged to receive a real-world description in the form of 
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an instruction set of a markup language, and said devices being operated 
according to said description. 

11. A system according to claim 10, wherein said devices are 
5 interconnected by a wireless network. 

12. A system according to claim 10, wherein said devices are 
interconnected by a powerline carrier network. 

10 13. A system according to any one of claims 10 to 12, wherein a 

device of said set is a local server. 

14. A system according to any one of claims 10 to 12, wherein said 
description is distributed amongst said devices. 

15 

15. A system according to any one of claims 10 to 12, wherein a 
device of said set comprises a user interface. 

16. A markup language enabled device comprising receiving means 
20 for receiving a real-world description in the form of an instruction set of a 

markup language and adjusting means for adjusting one or more parameters 
of the device according to said description. 

17. A device according to claim 16, wherein said device is an 
25 electronic device. 

18. A device according to claim 16 or 17, wherein said device is a 
fabric device. 

30 19. A device according to any one of claims 16 to 18, wherein said 

device is a display device. 
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20. A device according to any one of claims 16 to 19, wherein said 
device has audio capabilities. 

21. A device according to claim 16 or 17, wherein said device is a 
5 lighting device. 

22. A device according to claim 16 or 17, wherein said device is a 
heating device. 

10 23. A device according to any one of claims 16 to 22, wherein said 

receiving means includes part of a distributed browser stored on a record 
carrier, said part interpreting said instruction set and communicating with said 
adjusting means. 

is 24. A device according to claim 23, wherein said part of said browser 

is arranged to interpret instructions of a general type to generate specific 
parameter adjustments for communicating to said adjusting means. 

25. A markup language for describing real-world experiences 
20 comprising an instruction set including instructions interpretable by a markup 

enabled device to operate said device. 

26. A physical carrier having embedded therein executable 
instructions, said instructions forming part of the markup language of claim 25. 

25 

27. A computer-readable medium having executable instructions 
thereon, said instructions forming part of the markup language of claim 25. 

28. A computer-readable medium according to claim 27, wherein 
30 said medium is of a recordable format. 
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29. An instruction set comprising an executable subset of the markup 
language of claim 25. 

30. A computer program for authoring an instruction set of the 
5 markup language of claim 25. 

31. A computer program for interpreting an instruction set of the 
markup language of claim 25 into a set of device operations. 

10 32. A language according to claim 25, wherein said real-world 

experiences include visual, audio, olfactory and tactile sensations. 

33. A language according to claim 25, wherein said language is XML 
compliant. 
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